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PREFACE 


This  document  was  prepared  under  a  task  titled  “ESTCP/SERDP:  Assessment  of 
Traditional  and  Emerging  Approaches  to  the  Detection  and  Identification  of  Surface  and 
Buried  Unexploded  Ordnance.”  The  authors  would  like  to  thank  Drs.  Jeff  Marqusee  and 
Anne  Andrews  of  the  ESTCP/SERDP  Program  Office  for  their  guidance  in  carrying  out 
this  analysis  and  for  their  careful  review  of  the  report.  In  addition,  we  would  like  to 
express  our  gratitude  to  the  Oak  Ridge  National  Laboratory  and  Naval  Research 
Laboratory  teams  for  their  cooperation  in  this  effort.  Both  groups’  willingness  to  answer 
questions  concerning  their  systems,  the  way  they  were  operated,  and  the  data  collected 
made  the  analysis  job  easier  and  more  complete.  Their  reviews  of  the  draft  versions  of  the 
report  raised  excellent  questions  whose  treatment  in  the  final  version  have  made  this  a 
better  report. 
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EXECUTIVE  SUMMARY 


In  July  2002,  the  Naval  Research  Laboratory  (NRL)  Airborne  Multi-Sensor 
Towed  Array  Detection  System  (aMTADS)  and  the  Oak  Ridge  National  Laboratory 
Airborne  Geophysics  System  (ORAGS)  magnetometer  systems  surveyed  several  areas  at 
the  Aberdeen  Proving  Ground.  Topographies  and  land  cover  ranged  from  flat  and  grassy, 
treed  with  brush,  and  wetlands  to  shallow  ponds  and  marine  offshore  areas.  The  systems 
employed  similar  cesium-vapor,  total-field  magnetometers,  but  differed  in  platform  setup 
and  data  processing: 

•  Both  systems  employ  similar  Bell  helicopters  and  magnetometers  mounted 
on  booms.  aMTADS  mounts  seven  sensors  on  a  forward  boom.  ORAGS 
deploys  four  sensors  on  a  forward  boom  and  four  on  a  mid-ship  boom. 

•  The  two  teams  use  different  Global  Positioning  System  (GPS)-based  systems 
to  provide  location  and  attitude  data  and  different  altimeter  combinations  to 
provide  height.  Each  team  has  developed  its  own  algorithms  for  data 
calibration,  smoothing,  and  interference  reduction. 

•  The  Oak  Ridge  system  uses  automatic  target  recognition  algorithms  to 
process  the  magnetometer  data  and  classify  targets.  The  NRL  system  relies 
on  human  judgment  aided  by  computer-implemented,  dipole-fit  routines. 

Probability  of  detection,  background  alarm  rates,  and  other  appropriate  statistics 
were  calculated  for  aMTADS  and  ORAGS  through  two  major  methodologies: 

•  163  unexploded  ordnance  (UXO)  items  comprising  60  mm  and  81  mm 
mortars  and  105  mm  and  155  mm  shells  were  buried  among  intrinsic  clutter 
at  four  topographically  different  sites. 

•  A  dig  list  of  unknown  items  was  generated  using  data  from  both  systems,  and 
a  subset  of  the  unknown  items  was  subsequently  dug  and  identified. 

Each  system  provided  an  ordered  list  of  detections  and  assigned  each  detection  a 
“confidence  of  ordnance”  ranging  from  1  being  “high  confidence  UXO”  to  6  being  “high 
confidence  clutter.” 

A  priori  consideration  of  the  similarity  between  aMTADS  and  ORAGS  suggests 
that  they  should  perform  about  the  same.  However,  their  different  platform  setups, 
operating  parameters  (principally  altitude  and  speed),  and  data-processing  methods 
produced  different  results,  with  aMTADS  performing  better  than  ORAGS,  particularly  in 
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the  more  benign  environments.  Based  on  the  results,  we  have  drawn  the  following 
eonelusions: 

•  If  maximum  sensitivity  is  desired,  it  is  important  that  the  system  design  allow 
very  low-level  flight  in  areas  where  that  is  possible.  Careful  attention  to 
redueing  apparent  spatial  baekground  variations  eaused  by  platform 
dynamies  effeets  is  also  erueial  to  obtaining  maximum  deteetion  sensitivity. 
In  the  very  benign  airfield  terrain  and  baekground,  aMTADS  deteeted  49  of 
52  emplaeed  items  ineluding  all  three  60  mm  mortars  and  eighteen  of  twenty- 
one  81  mm  mortars,  while  ORAGS,  flying  about  a  meter  higher  and  three 
times  as  fast  and  with  four  times  the  baekground  standard  deviation,  deteeted 
14  fewer  emplaeed  items,  missing  1  of  the  60  mm,  1 1  of  the  81  mm  and  5  of 
the  105  mm. 

•  Sueeessful  deteetion  of  UXO  targets  in  bodies  of  water  will  require 
knowledge  of  the  burial  depth  and  type  of  ordnanee  expeeted.  Helieopter 
systems  sueh  as  these  already  have  the  disadvantage  of  a  sensor  that  is  a 
meter  or  two  above  the  surfaee.  Beeause  of  the  l/(distanee)^  signal  falloff, 
even  moderate  target  depths  make  the  deteetion  of  all  but  large  targets 
marginal. 

•  Neither  system  shows  signifieant  individual  target  deteetion  eapability  in 
highly  eluttered  areas. 

•  Target  loeation  aeeuraey  is  important  for  unambiguous  target  reeovery. 
aMTADS  demonstrates  that  average  radial  position  errors  of  less  than  40  em 
are  possible  from  an  airborne  system  using  real-time,  differential  GPS. 
However,  obtaining  that  degree  of  aeeuraey  requires  reliable  sensing  and 
preeise  aeeounting  for  helieopter  dynamie  effeets  on  sensor  position. 

•  Aeeurate  automatie  deteetion  and  elassifieation  algorithms  would  be  of 
signifieant  benefit  for  systems  (sueh  as  the  ones  demonstrated  here)  that  are 
eapable  of  surveying  hundreds  of  aeres  per  day.  Current  automatie 
algorithms,  however,  do  not  perform  as  reliably  as  human  analysts. 
Partieularly  in  the  deteetion  phase,  they  result  in  many  more  anomalies  that 
would  have  to  be  dug,  without  adequate  diserimination  eapability  to  make  up 
for  the  additional  deteetions. 

•  Current-generation  helieopter-borne  magnetometer  systems  are  eapable  of 
reliably  deteeting  areas  of  high  ferrous  elutter/UXO  density.  It  appears  they 
are  not  suitable  for  general  survey  use  to  deteet  individual  ordnanee  items, 
exeept  in  speeial  eonditions  that  provide  a  eombination  of  benign  terrain,  low 
baekground  elutter,  and  appropriate  UXO  sizes  (>60  mm).  Even  then,  lower 
and  slower  flight  than  indieated  in  system  speeifieations  will  likely  be 
required  to  aehieve  adequate  deteetion  performanee. 
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I.  INTRODUCTION 


The  purpose  of  this  teehnieal  report  is  to  provide  results  of  IDA’s  evaluation  of 
data  eolleeted  by  the  Naval  Researeh  Laboratory  (NRL)  and  the  Oak  Ridge  National 
Laboratory  (ORNL)  airborne  magnetometer  systems  in  testing  eondueted  at  Aberdeen 
Proving  Ground  (APG),  Maryland,  during  July  2002.  We  first  give  a  brief  deseription  of 
the  test  areas  and  the  data  eolleeted.  We  then  summarize  the  detailed  analysis  to  allow  the 
reader  to  understand  the  basis  upon  whieh  eonelusions  are  drawn.  ORNL  and  NRL  report 
on  this  operation  in  referenees  1  and  2.  Those  doeuments  provide  more  detail  on  the 
helieopter-mounted  systems  and  on  the  deteetion  and  elassifieation  proeessing.  Referenee 
3,  the  U.S.  Army  Aberdeen  Test  Center  report  on  the  surveys,  provides  additional  detail 
on  the  demonstration  areas  and  on  the  emplaeement  and  removal  of  the  seeded  ordnanee 
items. 

A,  APG  TEST  SITES  [3,  4] 

The  Aberdeen  testing  took  advantage  of  some  areas  within  the  proving  ground 
that  eontain  preexisting  ordnanee  items  and  serap,  as  well  as  several  areas  that  were 
seeded  with  ordnanee  from  the  Standardized  Test  Site  inventory.  A  ealibration  area  was 
established  near  an  on-base  airfield  historieally  elear  of  UXO,  and  10  ordnanee  items 
whose  loeations  and  orientations  were  known  to  the  demonstrators  were  buried  in  an  area 
thought  to  be  relatively  uneluttered.  Although  the  ealibration  area  generally  had  little 
elutter,  a  very  large  magnetie  anomaly  in  the  vieinity  of  several  of  the  ealibration  targets 
negated  their  usefulness  for  analysis.  In  addition,  an  open  field  area  in  the  vieinity  of  the 
airfield  was  seeded  with  52  ordnanee  items  in  loeations  unknown  to  the  demonstrators. 
Figure  1  is  an  aerial  photograph  of  the  airfield,  with  the  ealibration  (eheekout)  and  open 
field  areas  shown. 

The  remaining  areas  with  seeded  ordnanee  were  a  dewatering  pond  area  (47 
items);  an  aetive  reeovery  field  for  indireet-fire  weapons  (24  items),  whieh  also  eontained 
areas  of  phragmites  (8  items);  and  littoral  areas  (32  items).  Surveys  were  also  eondueted 
over  a  mine,  grenade,  and  direet-fire  weapon  range  and  over  a  Chesapeake  Bay  tidal 
water  area  that  is  an  old  impaet  area.  In  those  areas,  no  ordnanee  was  emplaeed  or  dug, 
and  no  seoring  was  done  due  to  the  high  density  of  UXO.  More  detail  eoneerning  the 
individual  sites  is  provided  in  the  analysis  seetion. 


I-l 


Figure  1.  Airfield  Area  with  the  Calibration  (Checkout)  and  Open  Field  Areas  Marked 
B,  SENSOR  PLATFORMS  [5,  6] 

Two  helicopter-borne  sensor  arrays  were  demonstrated  in  the  tests  at  APG.  Each 
is  carried  on  a  similar  Bell  206L  Long  Ranger  helicopter.  The  arrays  employ  a  cesium 
vapor,  total  field  magnetometer  (Geometries  Model  822  variants)  array  mounted  on  a 
nonmetallic  boom,  and  a  differential  Global  Positioning  System  (GPS)  to  provide 
accurate  position  data  for  anomaly  location  and  for  discrimination  algorithm  inversion. 
The  arrays  and  navigation  instrumentation,  however,  differ  in  a  number  of  details. 

The  Oak  Ridge  Airborne  Geophysics  System  (ORAGS),  developed  by  ORNL  and 
shown  in  Figure  2,  deploys  the  magnetometer  sensors  on  an  “Arrowhead”  array.  This 
array  includes  four  outboard  magnetometers,  two  on  each  side  of  the  aircraft,  mounted  on 
an  amidships  horizontal  boom,  and  four  magnetometers  spaced  horizontally  on  a  boom 
forward  of  the  helicopter  nose.  Horizontal  spacing  of  the  magnetometers  is  1.75  m. 
Although  this  arrangement  provides  a  very  wide  array,  in  flight  the  pilot  is  generally 
unable  to  see  the  wide  amidships  booms,  and  the  fore-aft  spacing  complicates  signal 
leveling  among  the  sensors.  The  ORAGS  data-collection  system  samples  at  a  1,200  Hz 
rate,  but  commonly  downsamples  to  120  Hz  for  analysis.  The  data  provided  to  IDA  were 
at  the  120  Hz  rate. 
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Figure  2.  Oak  Ridge  Airborne  Geophysics  System 


The  airborne  version  of  the  NRL  Multi-Sensor  Towed  Array  Deteetion  System 
(MTADS)  shown  in  Figure  3,  whieh  we  designate  airborne  Multi-Sensor  Towed  Array 
Deteetion  System  (aMTADS),  is  an  outgrowth  of  an  earlier  vehiele -towed  system.  It 
employs  seven  magnetometers  on  a  single  horizontal  boom  in  front  of  the  helieopter 
nose.  Magnetometer  spaeing  is  1.5  m,  and  sensor  sampling  is  done  at  100  Hz. 


Figure  3.  NRL  Airborne  Muiti-Sensor  Towed  Array  Detection  System 


Speeifieations  for  both  systems  eall  for  an  altitude  of  a  few  meters  above  the 
ground  and  a  speed  of  around  20  m/s.  Operation  at  the  nominal  speed  results  in  down- 
traek  sample  spaeings  of  15-20  em  for  the  two  systems,  whieh  is  mueh  finer  than  the 
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cross-track  sampling.  Cross-track  sampling  finer  than  the  1.5  m  or  1.75  m  sensor  spacing 
can  be  provided  by  multiple  or  overlapping  passes  at  the  expense  of  area  coverage  rate. 
For  example,  aMTADS,  with  its  9  m  array  width,  typically  flies  7.0  m  line  separations 
(the  design  specification  calls  for  7.5  m  line  separations).  In  addition,  in  the  airfield  area, 
aMTADS  speeds  were  generally  around  5  m/s  and  altitudes  were  closer  to  1  m.  ORAGS 
speed  and  altitude  for  those  measurements  were  closer  to  15  m/s  and  2  m.  Effects  caused 
by  differences  in  altitude  and  speed  are  discussed  where  they  affect  analyzed  results. 

C.  DATA  ANALYZED 

The  demonstrators  provided  two  types  of  data  to  IDA  for  analysis:  magnetometer 
data  and  target-detection  lists.  Raw  and  processed  magnetometer  data  were  provided  for  a 
high-altitude  flight  segment  and  for  both  the  airfield  sites.  Detection  lists  were  provided 
by  both  demonstrators  for  the  three  sites  containing  seeded  ordnance.  ORNL  provided 
two  lists.  One  employs  what  they  term  a  univariate  detection  and  discrimination 
algorithm  and  the  other  a  multivariate  algorithm.  The  two  lists  were  scored  independently 
for  both  detection/discrimination  performance  and  location-error  statistics.  These  lists 
included  NAD83  coordinates;  target  size,  moment,  inclination,  and  azimuth;  and  a 
number  indicating  UXO  likelihood.  The  standard  classification  system  specified  by  the 
Environmental  Security  Technology  Certification  Program  (ESTCP)  sorts  detections  into 
six  categories  from  1  to  6,  with  1  denoting  most  likely  UXO  and  6  denoting  most  likely 
clutter.  That  target-classification  hierarchy  was  used  to  establish  receiver  operating 
characteristic  (ROC)  curves  for  each  system. 
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II.  ANALYSIS 


A,  OVERVIEW 

Two  basic  types  of  analysis  were  undertaken  using  the  APG  data.  The  first  was  an 
instrument  sensitivity  analysis  that  utilized  high-altitude  magnetometer  data  and  data 
eolleeted  over  the  airfield  ealibration  and  open  field  areas.  Sensor  noise  floors  were 
established  for  the  high-altitude  ease,  where  only  internal  noise  and  platform  effeets 
should  be  evident.  Noise  was  also  ealeulated  for  low-altitude  operation,  where  small- 
seale  elutter  was  expeeted  to  raise  the  noise  floor.  Peak  signal  response  was  determined 
over  a  number  of  targets  to  assess  typieal  signal-to-noise  ratios  (SNRs)  for  various  target 
types.  Deteetion  statisties  were  based  on  the  emplaeed  ordnanee  in  the  airfield  open  field 
area,  the  dewatering  ponds,  and  the  aetive  reeovery  field  areas.  Analysis  is  deseribed  and 
results  are  presented  below  for  eaeh  of  the  measures  of  system  performanee  examined. 
Note  that  ROC  eurves  presented  are  in  terms  of  probability  of  deteetion  (Pd)  of  emplaeed 
items  versus  the  baekground  alarm  rate.  Without  full  exeavation  of  all  anomalies,  true 
ROC  eurves  of  Pd  versus  false-alarm  probability  or  false-alarm  rate  eannot  be  generated. 
For  the  airfield  open  field  area,  subsequent  examination  of  20  anomalies  from  the 
operators’  deteetion  lists  found  18  items  to  be  non-ordnanee-related  elutter  (mower  blade, 
serap  iron,  welding  rods,  ete.).  Two  items,  found  together,  were  inert  voleano  mines. 
Thus,  for  the  airfield,  whieh  was  historieally  elear  of  UXO,  the  limited  exeavations 
appear  to  indieate  that  the  baekground  alarms  are  heavily  dominated  by  non-ordnanee- 
related  elutter.  That  is  not  the  ease  for  the  other  areas. 

B,  SENSITIVITY 

The  goal  of  the  sensitivity  analysis  was  to  assess  the  noise  properties  of  the  two 
sensor  arrays  and  to  establish  the  SNR  levels  that  eould  be  aehieved  against  a  variety  of 
targets.  Ultimately,  SNR  is  a  funetion  of  a  number  of  system-  and  operation-related 
faetors.  These  inelude  the  inherent  sensitivity  and  internal  noise  eharaeteristies  of  the 
eesium  vapor  magnetometers  and  data-eolleetion  system  eleetronies,  the  ability  of  data 
proeessing  to  remove  platform  noise  and  large-seale  geologie  noise  without  suppressing 
signal  eomponents,  the  height  of  the  sensor  above  the  target,  and  the  proximity  of  the 
peak  data  point  to  the  aetual  peak  of  the  magnetie  response.  While  we  made  an  attempt  to 
understand  the  faetors  affeeting  relative  sensor  eapabilities,  the  foeus  of  this  effort  was  to 


assess  performance  based  on  the  data  delivered  by  the  demonstrators.  Because  the 
processed  data  were  used  to  make  detection  and  classification  decisions,  the  emphasis 
was  on  an  analysis  of  those  data;  however,  raw  magnetometer  data  gathered  in  high- 
altitude  flights  were  also  analyzed  to  understand  the  noise  added  by  the  platforms. 

The  first  analysis  undertaken  utilized  raw  and  processed  magnetometer  data  from 
the  high-altitude  flights  to  establish  a  fundamental  noise  floor  for  each  of  the  sensor 
systems.  Figure  4  provides  the  output  of  a  fast  Fourier  transform  (FFT)  of  raw,  de- 
medianed,  high-altitude  data  from  both  sensors.  The  composite  spectrum  was  produced 
by  taking  an  FFT  of  20  sec  of  each  magnetometer  time  series  and  then  summing  the 
spectral  magnitudes.  The  data  are  displayed  in  decibel  space  to  allow  the  data  dynamic 
range  to  be  better  visualized,  and  each  abscissa  is  scaled  to  the  highest  unambiguous 
frequency  provided  by  the  sample  rate  (60  Hz  for  ORAGS  and  50  Hz  for  aMTADS).  No 
attempt  was  made  to  normalize  the  response  between  the  two  systems,  so  the  absolute 
levels  on  the  two  plots  are  meaningless.  On  both  plots,  the  strong  signal  near  6.5  Hz  and 
its  harmonics  are  due  to  the  helicopter  blades.  Most  of  the  target  information  lies  at 
frequencies  well  below  the  blade  modulation  signals,  and  one  major  purpose  of  data 
preprocessing  is  to  remove  the  high-frequency  platform  interference  without  removing 
the  target  signal. 

Figure  5  provides  the  ORNL  data  record  from  which  the  ORAGS  spectrum  above 
was  produced.  The  upper  figure  provides  the  time  history  of  each  of  the  magnetometers, 
and  the  bottom  figure  plots  the  same  data  after  a  de -median  filter  has  been  applied.  Only 
the  20-second  segment  between  15  and  35  seconds  was  used  in  the  subsequent  analysis, 
not  only  because  it  visually  represents  a  relatively  constant  segment  of  the  data,  but 
because  it  also  contains  enough  points  to  provide  good  statistics.  Figure  6  provides 
histograms  of  the  raw,  de-medianed,  high-altitude  data  from  that  segment  and  the  same 
data  after  ORNL  processing.  Note  that  processing,  which  basically  applies  filtering  to 
remove  high-frequency  components,  significantly  reduces  data  spread.  For  the  high- 
altitude  data,  the  noise  signal  is  nominally  Gaussian,  as  expected,  and  has  a  standard 
deviation  of  0.56  nT.  Processing  reduces  the  standard  deviation  to  0.14  nT. 
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Figure  4.  Sensor  Spectral  Output  from  Raw,  High-Altitude  Data 
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Figure  5.  Oak  Ridge  Airborne  Geophysics  System  High-Altitude  Raw  Data  and  After 

Passage  through  a  De-median  Filter 

Figure  7  provides  approximate  altitude  (height  above  ellipsoid,  HAE)  and  signal 
data  for  the  aMTADS  system,  where  output  from  eaeh  of  the  magnetometer  sensors  is 
plotted.  Note  that  only  six  magnetometer  signals  are  provided  in  the  plots.  Beeause  one 
magnetometer  was  not  funetioning  correetly  during  the  high-altitude  calibrations,  its 
signal  was  not  used  in  the  high-altitude  noise  analysis.  The  HAE  data  are  included 
because  NRL  chose  to  fly  a  relatively  complex  high-altitude  pattern  that  included  course 
changes  and  maneuvers.  The  maneuvers  are  evident  in  the  HAE  data  after  622  seconds, 
which  show  different  sensor  altitudes  due  to  helicopter  banking.  To  provide  a 
representative  sample  for  straight  flight,  only  the  raw  data  from  600  to  620  seconds  were 
used  in  the  analysis.  The  raw  data  contain  a  long-term  trend  and  some  differences 
between  the  individual  sensor  channels.  As  with  the  ORAGS  data,  IDA  removed  these  by 
applying  a  de-median  filter  to  each  channel.  Figure  8  provides  a  histogram  of  the  raw 
high-altitude  magnetometer  data  after  the  de-median  operation  and  after  NRL  signal 
processing.  The  noise  signal  is  nominally  Gaussian,  as  expected,  and  has  a  standard 
deviation  of  1 .36  nT. 
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Figure  6.  Histograms  of  Oak  Ridge  Airborne  Geophysics  System  High-Aititude 
De-medianed  Data  and  Processed  Data 
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Figure  7.  Raw  High-Altitude  Data  from  Airborne  Multi-Sensor  Towed 
Array  Detection  System 

The  processed  high-altitude  data  histogram  in  Figure  8  provides  a  standard 
deviation  of  0.5  nT.  Subsequent  analysis  of  quiet  areas  in  the  airfield  calibration  and  open 
field  areas  yielded  aMTADS  standard  deviation  values  as  low  as  0.09  nT.  Other  than 
effects  of  aircraft  maneuvers  and  speed,  there  is  no  physical  reason  the  high-altitude  data 
should  be  noisier  than  near-ground  data,  so  we  investigated  the  spectrum  for  the 
processed  high-altitude  data.  Residual  energy  in  the  6.5  Hz  region  was  clearly  visible  in 
the  spectrum,  indicating  that  the  preprocessing  used  for  the  high-altitude  data  had  not 
removed  all  the  blade  noise.  In  its  standard  processing,  NRL  applies  both  a  notch  filter 
and  a  low-pass  fdter  to  the  raw  data  [2].  We  crudely  simulated  the  effects  of  low -pass 
filtering  by  zeroing  the  FFT  spectral  bins  above  6.45  Hz  and  inverse  transforming  the 
complex  spectral  data  to  recover  a  time  series.  After  the  crude  low-pass  filtering 
operation,  the  aMTADS  processed  data  achieve  a  0.16  nT  standard  deviation,  almost 
equal  to  that  seen  with  ORAGS.  Although  this  value  is  still  larger  than  seen  in  some  of 
the  near-ground  quiet  areas,  we  attribute  the  difference  to  our  crude  filtering  scheme. 
Note  that  the  high-altitude  noise  data  were  not  used  in  any  subsequent  signal-to-noise 
ratio  analysis.  For  those  analyses,  the  low-altitude  noise  in  the  local  area  is  a  more 
realistic  measure  of  system  performance.  The  high-altitude  noise  analysis  was  simply 
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done  as  a  sanity  check  and  to  ensure  that  we  understood  the  data  provided  to  IDA.  All  the 
scoring  analysis  used  the  processed  data  as  it  was  delivered  by  the  two  demonstrators. 
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Figure  8.  Histograms  of  Airborne  Muiti-Sensor  Towed  Array  Detection  System  High- 
Aititude  De-medianed  Data  and  Processed  Data 
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Having  established  the  noise  floors  for  the  sensors,  the  remaining  question  to  be 
answered  regarding  sensitivity  was  what  target  signal  eould  be  achieved  over  various 
targets  and  thereby  what  SNR.  Because  it  was  likely  that  noise  performance  would 
degrade  close  to  the  ground,  where  small  ferrous  clutter  items  and  fine-scale  geology 
could  raise  the  noise  floor,  the  noise  used  to  reference  SNR  was  that  from  a  visually  quiet 
area  in  the  vicinity  of  the  targets. 

The  first  area  assessed  for  SNR  performance  was  the  calibration  lane  area  within 
the  airfield.  That  area  contained  two  each  of  five  ordnance  types,  ah  horizontal,  with  one 
buried  facing  north-south  and  one  east-west.  Figure  9  shows  ORAGS  and  aMTADS 
magnetometer  maps  of  the  calibration  area.  Target  locations  are  indicated  by  an  x  for  the 
60  mm,  and  going  from  northwest  to  southeast,  the  remaining  targets  in  pairs  denoted  by 
circles  of  different  colors  are  2.75  inch,  81  mm,  105  mm,  and  155  mm.  Data  from  the  box 
at  the  top  of  the  area  were  used  to  establish  the  noise  floor  for  each  system. 

Note  that  both  figures  contain  a  bright  anomaly  centered  at  approximately  -100  N 
and  -50  E.  That  anomaly  prevented  analysis  of  the  east-most  81  mm  target.  In  addition, 
no  peak  could  be  chosen  in  the  ORAGS  data  for  the  one  of  the  two  60  mm  mortar  targets 
or  in  either  data  set  for  one  of  the  2.75-inch  rounds.  Table  1  provides  peak  magnetometer 
levels  from  the  data  for  each  of  the  targets  that  could  be  analyzed.  Two  columns  of  peak 
values  are  provided  for  each  system.  In  each  case  the  left  column  is  for  the  more 
northwesterly  item  of  each  ordnance  pair  in  Figure  9,  and  the  right  column  is  for  the  more 
southeasterly  item. 

Individual  peak  signatures  for  a  given  ordnance  type  can  vary  significantly 
because  of  ordnance  orientation  and  also  because  of  possible  offsets  (with  1.5  m  or  1.75 
m  cross-range  sensor  spacing)  of  the  nearest  magnetometer  from  the  signal’s  actual  peak. 
Nevertheless,  a  reasonable  sample  of  items  should  be  expected  to  average  the  offset 
potential  among  the  sensors,  and  in  any  event,  the  SNR  achieved  is  one  legitimate 
measure  of  how  a  system  actually  performed.  In  this  case,  if  only  common  ordnance 
items  are  considered,  the  average  peak  signal  seen  by  aMTADS  is  1.8  times  larger  than 
the  peak  signal  seen  by  ORAGS. 
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Figure  9.  Oak  Ridge  Airborne  Geophysics  System  and  Airborne  Muiti-Sensor  Towed  Array 
Detection  System  Magnetometer  Data  for  the  Airfield  Calibration  Area 
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Table  1.  Magnetometer  Peaks  from  the  Airfield  Calibration  Area 


Target  Pair 
Type 

ORAGS 

Peak (nT) 

aMTADS 

Peak (nT) 

60  mm 

2.44 

- 

8.78 

3.82 

2.75  inch 

5.60 

- 

9.84 

- 

81  mm 

3.89 

- 

11.32 

- 

105  mm 

11.93 

55.79 

41.69 

49.75 

155  mm 

19.23 

43.65 

56.51 

85.17 

To  convert  peak  signal  to  peak  SNR,  the  quiet  area  in  the  box  was  analyzed  to 
establish  a  background  noise  level  for  each  system.  Figure  10  provides  blow-ups  of  the 
quiet  area  for  eaeh  system,  with  seales  ehosen  to  emphasize  small  variations  in  signal.  In 
Figure  10,  the  ORAGS  return  shows  a  pattern  on  a  seale  unlikely  to  be  eonneeted  with 
small  elutter  anomalies,  while  the  aMTADS  return  is  mueh  more  eonstant,  with  the 
exeeption  of  what  may  be  a  small  anomaly  in  the  northwest  eorner.  Figure  1 1  shows  the 
proeessed  data  histograms  over  that  area,  where  the  differenee  in  noise  eharaeteristies  is 
even  more  obvious.  The  aMTADS  system  shows  a  noise  distribution  that  is 
approximately  Gaussian,  albeit  with  some  asymmetry,  and  a  standard  deviation  of 
0.09  nT.  ORAGS,  on  the  other  hand,  shows  a  deeidedly  non-Gaussian  histogram.  The 
proeessed  data  that  IDA  analyzed  were  provided  in  a  map  format  (i.e.,  a  data  array  that 
gave  only  signal  amplitude  and  position)  and  not  keyed  to  the  partieular  sensor  in  the 
array  that  eolleeted  the  data.  Part  of  the  manipulation  to  transform  raw  magnetometer 
data  into  proeessed  data  is  to  level  the  sensor  outputs.  The  map  of  the  quiet  area  and  the 
shape  of  the  ORAGS  histogram  imply  that  leveling  was  not  perfeet,  and  the  density 
funetion  appears  to  be  the  result  of  summing  several  non-zero-mean  Gaussians.  Beeause 
of  that,  the  noise  standard  deviation  in  the  quiet  area  for  ORAGS  was  0.51  nT. 

Using  the  baekground  noise  measured  for  eaeh  sensor  array  and  the  peaks  of 
Table  1,  SNR  values  were  ealeulated  for  the  ordnanee  items  analyzed.  Those  values  are 
shown  in  Table  2.  The  eombination  of  the  lower  sensor  noise  level  and  the  larger  signal 
gives  the  aMTADS  about  a  faetor  of  10  advantage  in  peak  SNR  level  on  eomparable 
targets. 
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Figure  10.  Magnetometer  Maps  for  the  Airfield  Calibration  Lane  Quiet  Area 
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Figure  11.  Histograms  of  the  Magnetometer  Outputs  for  the  Airfieid  Caiibration  Lane  Quiet 

Area  for  (a)  DRAGS  and  (b)  aMTADS 
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Table  2.  Peak  SNR  Values  from  the  Airfield  Calibration  Area 


Target  Pair 
Type 

ORAGS 

SNR 

aMTADS 

SNR 

60  mm 

4.8 

- 

97.6 

42.4 

2.75  inch 

11.0 

- 

109.3 

- 

81  mm 

7.6 

- 

125.8 

- 

105  mm 

23.4 

109.4 

463.2 

552.8 

155  mm 

37.7 

85.6 

627.9 

946.3 

We  felt  some  eoneern  limiting  the  SNR  analysis  to  the  ealibration  area, 
partieularly  beeause  mueh  of  the  SNR  differenee  was  a  result  of  a  lower  noise 
baekground  for  the  aMTADS.  Therefore,  additional  SNR  analysis  was  aeeomplished  on 
seleeted  targets  in  the  airfield  open  field  area.  Figure  12  provides  magnetometer  maps  for 
the  ORAGS  and  aMTADS  eolleetions  at  that  site,  with  the  six  items  ehosen  for  further 
SNR  analysis  denoted  by  arrows  and  numbers.  The  eriteria  used  for  ehoosing  the  items 
were  that  they  show  a  elear  peak  in  both  data  sets  and  that  they  lie  in  an  area  where  there 
appears  to  be  no  eontamination  of  the  target  signal  by  other  anomalies.  No  60  mm  targets 
were  ehosen.  Table  3  provides  the  signal  levels  for  the  6  targets  analyzed. 


Table  3.  Magnetometer  Peaks  from  the  Airfield  Open  Field  Area 


Target  # 

Target  Type 

ORAGS  (nT) 

aMTADS  (nT) 

aMTADS/ORAGS 

1 

105  mm 

32.33 

172.86 

5.35 

2 

105  mm 

33.60 

123.16 

3.67 

3 

105  mm 

8.20 

31.99 

3.90 

4 

81  mm 

4.79 

24.21 

5.05 

5 

81  mm 

6.74 

24.77 

3.68 

6 

81  mm 

5.61 

5.73 

1.02 

With  the  exeeption  of  one  81  mm  mortar,  the  peak  aMTADS  signals  were  three  to 
five  times  those  of  the  ORAGS  system.  A  potential  reason  for  the  higher  aMTADS  signal 
levels  is  the  higher  data  density  provided  by  that  system  in  this  survey.  In  one  airfield 
area  analyzed,  the  ORAGS  data  provided  about  4  points/m^.  This  is  slightly  denser  than 
the  3.4  points/m^  that  would  be  ealeulated  for  the  array  spaeing  and  sample  rate.  In 
eontrast,  aMTADS  provided  21  points/m^,  signifieantly  more  than  the  3.3  points/m^  that 
would  be  ealeulated  for  its  sensor  spaeing  and  sample  rate.  Further  investigation  and 
diseussions  with  the  NRL  team  showed  that  the  higher  density  was  partially  a  result  of 
the  normal  7.0  m  line  spaeing  but  mostly  due  to  redueed  helieopter  speed. 
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Figure  12.  Oak  Ridge  Airborne  Geophysics  System  and  Airborne  Muiti-Sensor  Towed 
Array  Detection  System  Magnetometer  Maps  of  the  Airfieid  Open  Fieid  Area 
(Empiaced  items  are  indicated  by  A  for  105  mm,  o  for  81  mm,  and  x  for  60  mm) 
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Increased  data  point  density  implies  decreased  point  spacing  down-range,  cross¬ 
range,  or  both.  In  this  case,  the  7.0  m  line  spaeing  provides  potentially  deereased  eross- 
range  data  spaeing  (if  the  sensor  paths  do  not  align  in  the  overlap  region,  whieh  in 
general  they  did  not),  and  the  slower  speed  provides  deereased  down-range  spaeing.  Even 
at  normal  helieopter  speed,  the  eross-range  and  down-range  point  separations  are  so 
different  (1.5-1.75  m  in  eross-range  vs.  0.15-0.2  m  in  down-range)  that  the  potential 
benefits  of  deereasing  eaeh  are  different. 

The  major  potential  benefit  of  deereased  eross-range  spaeing  is  inereased 
probability  that  a  data  point  will  he  on  the  peak  of  the  target  return.  For  a  helieopter  at  a 
2  m  altitude,  a  point  target  at  the  surfaee,  and  an  assumed  l/(distanee)^  signal  falloff,  the 
peak  signal  provided  by  a  target  direetly  under  an  ORAGS  sensor  would  be  1.3  times 
stronger  than  one  with  maximum  separation  from  the  sensor.  For  aim  helieopter 
altitude,  the  improvement  is  a  faetor  of  2.3.  For  extended  targets,  however,  the  effeet 
would  be  deereased.  Thus,  eross-range  sample  density  ean  be  only  a  part  of  the  reason  for 
the  stronger  aMTADS  signal. 

Beeause  the  spatial  signal  is  signifieantly  oversampled  in  the  down-range 
direetion,  even  at  normal  helieopter  speeds,  the  advantages  of  eloser  sample  spaeing  in 
that  dimension  are  more  subtle.  Nevertheless,  an  advantage  does  arise  beeause  of  the 
relationship  between  the  spatial  frequeneies  in  whieh  the  target  information  lies  and  the 
temporal  frequeneies  of  the  major  interferenee  souree,  the  helieopter  blade  rate  and  its 
harmonies.  With  the  aMTADS  100  Hz  sample  rate  (50  Hz  Nyquist  fequeney),  the  6.5  Hz 
blade  rate  interferenee  will  always  appear  in  a  Doppler  bin  that  is  13%  of  the  total 
unaliased  frequeney  band  from  the  zero  Doppler  bin.  In  eontrast,  deereasing  the  down- 
range  sample  spaeing  inereases  the  maximum  unambiguous  spatial  frequeney  in 
proportion  to  the  deereased  spaeing.  Thus,  the  blade  interferenee  appears  at  an  apparently 
higher  spatial  frequeney  for  deereased  sample  spaeing,  and  filtering  to  reduee  that 
interferenee  ean  be  more  effeetive  with  less  effeet  on  the  spatial  frequeneies  that  provide 
target  information.  Beeause  the  benefits  would  strongly  depend  on  the  level  of  the  blade 
interferenee,  the  details  of  the  filtering  used  to  suppress  it,  and  the  spatial  frequeneies  of 
the  target  data,  we  have  not  attempted  a  quantitative  analysis  of  performanee 
improvement  given  by  redueed  down-range  sample  spaeing.  While  it  is  unlikely  a  major 
effeet,  we  point  it  out  as  an  advantage  of  slower  speed  operation. 

Although  point  spaeing  ean  eertainly  have  some  effeet  on  relative  signal 
strengths,  helieopter  altitude  has  a  major  effeet  beeause  of  the  l/(distanee)^  signal  falloff 
In  the  table  below  are  estimates  for  the  parameters  of  the  dipoles  for  five  of  the  six  NRF 
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and  ORNL  targets  from  the  airfield  open  field  area  analyzed  in  Table  3.  (The  sixth  target 
was  not  analyzed  beeause  of  its  small  peak  amplitude  for  both  systems.)  To  obtain  these 
estimates,  the  NRL  and  ORNL  proeessed  data  were  interpolated  to  25  em  resolution,  a 
Matlab  image  was  eonstrueted  using  the  interpolated  data,  a  polygonal  region  was 
seleeted  in  the  image  to  isolate  a  dipole,  and  this  region  was  input  to  AETC  Ine.’s  dipole¬ 
fitting  routine.  The  routine  requires  estimates  of  the  baekground  field,  so  we  used  an 
online  resouree,  http://nssde.gsfe.nasa.gov/spaee/model/models/igrfhtml  to  provide  data 
for  the  magnitude  and  direction  of  the  sea-level,  background  magnetic  field  at  Aberdeen, 
Maryland.  Tables  4  and  5  provide  the  location  estimates  from  the  dipole-fit  routine  and 
the  ground  truth  values  for  the  five  targets  analyzed.  For  both  sets  of  data,  the 
background  magnetic  field  input  was  Bt  =  52697.5  nT,  Inclination  =  58.5°,  Declination  = 
3.6°.  To  assess  the  sensitivity  of  the  results  to  the  background  field,  we  estimated  target 
parameters  with  a  different  background  (Pecos,  Texas).  While  the  estimated  inclination 
and  declination  of  the  target  changed,  the  position  and  depth  estimates  did  not  change 
significantly. 


Table  4.  Ground-Truth  and  Dipole  Parameter  Estimates  Obtained  from  NRL  Processed 
Data  Using  Background  Field  for  Aberdeen,  Maryland,  at  Sea  Level 


ID 

Dipole  Parameter  Estimates 

Truth 

X 

Y 

Z 

East 

North 

Depth 

105MM-2 

-111.28 

-305.47 

1.39 

-111.30 

-305.39 

0.46 

105MM-4 

-107.36 

-330.76 

1.55 

-107.36 

-330.51 

0.46 

105MM-14 

-156.95 

-321.84 

1.23 

-157.28 

-321.36 

0.46 

81MM-10 

-149.77 

-326.65 

1.22 

-150.08 

-326.23 

0.11 

81MM-8 

-112.34 

-336.89 

1.53 

-112.31 

-336.30 

0.11 

Table  5.  Ground-Truth  and  Dipole  Parameter  Estimates  Obtained  from  ORNL  Processed 
Data  Using  Background  Field  for  Aberdeen,  Maryland,  at  Sea  Level 


ID 

Dipole  Parameter  Estimates 

Truth 

X 

Y 

Z 

East 

North 

Depth 

105MM-2 

-111.03 

-305.79 

2.55 

-111.30 

-305.39 

0.46 

105MM-4 

-107.06 

-331.11 

2.34 

-107.36 

-330.51 

0.46 

105MM-14 

-157.24 

-321.87 

1.86 

-157.28 

-321.36 

0.46 

81MM-10 

-149.40 

-326.72 

1.93 

-150.08 

-326.23 

0.11 

81MM-8 

-111.54 

-336.98 

2.14 

-112.31 

-336.30 

0.11 
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The  Z  estimate  from  the  dipole-fit  routine  is  distanee  from  the  sensor  to  the  target. 
From  the  ground  truth,  we  know  target  depth  and  ean  estimate  helieopter  altitude, 
assuming  that  a  sensor  flew  direetly  over  the  target.  Table  6  provides  an  estimate  of  the 
helieopter  altitude  for  eaeh  target,  along  with  the  faetor  by  whieh  the  two  signal 
amplitudes  should  differ  for  a  l/(distanoe)^  relationship.  The  averages  of  the  estimated 
signal  ratios  and  the  measured  signal  ratios  differ  by  less  than  10%.  Thus,  it  appears  that 
the  major  portion  of  the  differenee  in  signal  ean  be  explained  by  the  differenee  in 
operating  altitude.  It  is  not  surprising  that  basie  system  signal  sensitivities  should  be  very 
similar,  as  they  are  using  the  same  eesium  vapor  magnetometer  and  there  is  no  reason 
differenees  in  eleetronies  should  markedly  affeet  the  signal  amplitude. 

Table  6.  Estimated  Helicopter  Altitudes  and  Signal  Differences  Based  on  a  (distance)'^ 

Signal  Amplitude  Model 


ID 

aMTADS  alt. 
estimate  (m) 

DRAGS  alt. 
estimate  (m) 

Estimated 
signal  ratio 

Measured 
signal  ratio 

105MM-2 

0.93 

2.09 

6.17 

5.35 

105MM-4 

1.09 

1.88 

3.44 

3.67 

105MM-14 

0.77 

1.40 

3.46 

3.90 

81MM-10 

1.11 

1.82 

3.96 

5.05 

81MM-8 

1.42 

2.03 

2.74 

3.68 

Average 

1.06 

1.84 

3.95 

4.33 

Because  of  the  laek  of  terrain  relief  or  cover  in  the  airfield  area,  both  aircraft 
averaged  altitudes  lower  than  the  nominal  2  m  called  out  in  the  specifications.  aMTADS 
took  advantage  of  the  very  flat  terrain  to  fly  about  a  meter  off  the  ground.  If  such  low 
altitudes  can  be  safely  flown,  and  geology  and  clutter  are  benign,  these  data  show  that 
targets  as  small  as  60  mm  can  be  detected.  However,  the  very  rapid  fall-off  of  signal  with 
distance  means  that  small  increases  in  altitude  are  likely  to  significantly  reduce  detection 
capabilities  for  small  targets. 

Because  signal  is  only  one  part  of  SNR,  analysis  of  the  noise  levels  for  the  two 
systems  in  the  open  field  area  was  also  required.  The  selection  of  a  quiet  area  in  the 
airfield  open  field  area  was  not  as  straightforward  as  in  the  calibration  area  because  of 
obvious  small  clutter  peaks  over  much  of  the  open  field  survey  area.  A  tradeoff  must  be 
made  between  a  sufficient  number  of  points  to  provide  good  statistics  and  an  area  small 
enough  to  exclude  unwanted  anomalies.  To  assess  noise  level,  several  apparently  quiet 
areas  were  chosen,  then  all  points  within  an  increasing  radius  of  the  center  point  were 
included  in  the  noise  calculation.  In  general,  this  provided  a  relatively  stable  noise 
standard  deviation  until  an  anomaly  was  reached  that  rapidly  increased  the  value  of  the 
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standard  deviation.  Figure  13  shows  blow-ups  of  maps  produeed  by  eaeh  system  for  one 
of  the  quiet  areas  ehosen.  Visually,  this  appears  to  be  the  quietest  portion  of  the  airfield 
open  field  area  for  both  systems;  other  areas  provided  higher  noise  standard  deviations 
for  the  systems  that  were  obviously  affeeted  by  loeal,  small  anomalies. 

Figure  14  provides  eurves  for  the  signal  standard  deviation  as  a  funetion  of  radial 
distanee  from  the  referenee  point  for  both  systems.  As  the  radius  inereases,  all  points 
lying  within  the  enelosing  eirele  are  used  to  ealeulate  the  noise  standard  deviation.  In  this 
ease,  ORAGS  shows  a  monotonie  inerease  in  standard  deviation  vs.  radius  until  in  the 
vieinity  of  the  small  anomaly  around  17  m.  This  behavior  may  be  symptomatie  of  offsets 
aeross  the  array.  On  the  other  hand,  aMTADS  shows  a  low  but  relatively  stable  noise 
floor  that  inereases  only  as  an  obvious  anomaly  is  reaehed. 

Using  the  best  ease  results  for  ORAGS,  those  for  the  5  m  radius  eirele,  the  noise 
standard  deviation  for  ORAGS  is  0.36  nT  and  for  aMTADS  is  0.09  nT.  Based  on  those 
values  and  the  peak  signals  of  Table  3,  Table  7  gives  SNR  values  for  the  targets  in  the 
airfield  open  field  area. 

These  results  are,  in  general,  eonsistent  with  those  of  the  airfield  ealibration  area. 
That  is,  the  aMTADS  provides  a  noise  floor  that  is  lower  and  peak  signals  that  are  higher 
than  the  ORAGS,  resulting  in  an  SNR  differenee  of  nearly  a  faetor  of  17.  These  apparent 
sensitivity  differenees  are  refieeted  in  the  deteetion  performanee  deseribed  in  the  next 
seetion. 

C.  DETECTION 

The  deteetion  analysis  for  APG  foeused  on  assessing  the  pereentage  of  emplaeed 
ordnanee  items  deteeted  in  eaeh  of  the  areas  where  they  had  been  seeded.  Demonstrators 
were  required  to  bin  deteetions  into  six  eategories,  from  1  to  6,  with  1  eorresponding  to 
most  likely  UXO  and  6  eorresponding  to  most  likely  elutter.  This  binning  allows  a  ROC 
eurve  to  be  formed.  For  the  airfield  open  field  area,  we  have  ehosen  to  plot  probability  of 
deteetion  on  the  ordinate  and  use  baekground  alarm  rate  as  the  abseissa  for  two  reasons. 
First,  APG  is  fairly  eonfident  that  all  intaet  ordnanee  was  eleared  from  the  area,  so 
deteetions  not  eorresponding  to  emplaeed  ordnanee  are  likely  elutter.  The  limited  dig  at 
the  airfield  supports  that  assumption.  Seeond,  this  normalizes  the  deteetion  eurves.  That 
is,  a  demonstrator  ean  aehieve  a  probability  of  deteetion  of  1.0  simply  by  having  a  dense 
enough  set  of  deteetion  ealls.  (From  a  praetieal  point  of  view,  the  performanee  of  sueh  a 
system  might  be  judged  mueh  poorer  than  that  of  another  system  with  a  somewhat 
smaller  probability  of  deteetion,  but  mueh  lower  baekground  alarm  rate.) 
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Figure  13.  Magnetometer  Maps  of  One  Airfield  Quiet  Area 
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Statistics  for  "Quiet  Area"  centered  at  (y,  x)  =  (-300,  -75) 


Figure  14.  Noise  Standard  Deviation  as  a  Function  of  the  Radius  of  the  Area  Considered 

for  the  Quiet  Area  of  Figure  13 


Table  7.  Peak  SNR  Values  from  the  Airfield  Open  Field  Area 


Target  # 

Target  Type 

DRAGS  SNR 

aMTADS  SNR 

1 

105  mm 

89.8 

1920.7 

2 

105  mm 

93.3 

1368.4 

3 

105  mm 

22.8 

355.4 

4 

81  mm 

13.3 

269.0 

5 

81  mm 

18.7 

275.2 

6 

81  mm 

15.6 

63.7 

In  fact,  a  demonstrator  can  achieve  a  100%  Pd  without  benefit  of  a  survey.  All 
one  has  to  do  is  select  detection  locations  such  that  the  halos  used  for  grading  detection 
cover  the  entire  area.  For  uniform  randomly  placed  detection  calls  and  target  locations, 
we  might  expect  to  detect,  by  chance,  a  fraction  of  the  targets  given  by  ratio  of  the  area 
covered  by  the  halos  to  the  surveyed  site  area.  Thus,  in  general,  some  portion  of  the 
detections  credited  to  a  demonstrator  will  be  “true”  detections  and  some  will  be  “chance” 
[7].  Appendix  A  provides  an  analysis  that  allows  us  to  separate  detections  into  the  two 
categories.  After  a  discussion  of  the  detection  performance  of  both  demonstrators,  we 
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apply  the  true  versus  ehanee  deteetion  analysis  to  provide  better  understanding  of  the 
results. 


To  evaluate  deteetion  performanee,  a  eirele  (“halo”)  of  a  ehosen  radius  was 
ereated  around  eaeh  of  the  emplaeed  ordnanee  item  loeations.  If  a  system  deteetion 
deelaration  fell  on  or  within  the  eirele,  that  item  was  eounted  as  deteeted.  While  the  term 
“halo”  strietly  eonnotes  the  eireumferenee  of  the  eirele,  we  instead  use  the  term  here  to 
refer  to  the  eirele  radius,  as  that  is  the  direet  measure  of  the  maximum  loeation  error 
allowed  for  a  deteetion  to  be  eounted.  Halos  from  0.5  m  to  2  m  were  explored,  but  based 
on  the  target  loeation  aeouraeies  of  the  two  systems,  most  of  the  results  presented  are  for 
a  1.5  m  halo.  Where  illustrative,  results  for  1  m  and  2  m  halos  are  also  given.  Figure  15 
provides  the  ROC  eurves  based  on  a  1.5  m  halo  for  eaeh  system  in  the  airfield  open  field 
area.  Two  eurves  are  provided  for  the  ORAGS  system.  As  noted  earlier,  ORNL  personnel 
used  two  different  sets  of  algorithms  for  deteetion  and  diserimination  ealeulations.  The 
eurve  labeled  ORAGS -UV  and  ORAGS -MV  eorrespond  to  results  from  univariate  and 
multivariate  deteetion  and  elassifieation  algorithms,  respeetively.  In  this  ease,  the 
univariate  algorithm  performed  somewhat  better  than  the  multivariate.  For  both 
algorithms,  eategory  5  and  6  ealls  ineluded  no  additional  ordnanee,  and  approximately 
half  the  deteetion  ealls  were  in  those  eategories.  The  aMTADS  showed  substantially 
better  performanee  in  this  ease  than  either  ORAGS  algorithm.  However,  that 
performanee  ineluded  5  of  49  total  ordnanee  detections  in  category  5  and  1  in  category  6. 


Figure  15.  Receiver  Operating  Characteristics  Curves  for  the 
Airfieid  Open  Fieid  Area  for  a  1.5  m  Haio 
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In  addition  to  calculating  the  ROC  curves  for  the  airfield,  we  analyzed  the  type  of 
ordnance  detected  as  a  function  of  the  halo  size  for  1  m,  1.5  m,  and  2  m  halos.  Table  8 
provides  those  results.  In  going  from  a  1  m  to  a  2  m  halo,  only  the  ORAGS-MV  results 
showed  a  substantial  improvement  in  detection,  rising  from  15  percent  below  the 
univariate  algorithm  results  to  2  percent  above  over  that  range.  This  is  not  surprising,  due 
to  the  east  bias  of  23.3  cm  and  a  north  bias  of  39.7  cm  in  position  estimation  shown  by 
the  multivariate  algorithm,  compared  with  0.7  cm  east  and  6.4  cm  north  for  the  univariate 
algorithm  and  -6.1  cm  east  and  8.4  cm  north  for  aMTADS  (see  Figure  16).  Based  on  the 
chance  detection  analysis  for  this  site,  discussed  later,  the  curves  in  Figure  15  for  the 
1.5  m  halo  appear  to  be  representative  of  relative  system  performance. 


Table  8.  Ordnance  Detection  Results  for  the  Airfield  Open  Field  Area  for 

Three  Detection  Halos 


Ordnance 

Emplaced 

ORAGS 

aMTADS 

(1  m) 
UV/MV 

(1.5  m) 
UV/MV 

(2  m) 
UV/MV 

(1  m) 

(1.5  m) 

(2  m) 

60  mm 

3 

2/1 

2/2 

2/2 

3 

3 

3 

81  mm 

21 

9/7 

10/10 

12/13 

16 

18 

18 

105  mm 

28 

21/16 

23/21 

23/23 

27 

28 

28 

Total 

52 

32/24 

35/33 

37/38 

46 

49 

49 

We  also  created  location-error  scatter  plots  for  the  1.5  m  halo.  Figure  16  provides 
those  data  for  each  system.  (We  don’t  show  the  ORAGS-MV  algorithm  performance, 
because  it  was  worse  than  that  of  the  ORAGS-UV  algorithm.)  The  ORAGS-UV 
detections  provided  east  and  north  error  standard  deviations  of  42.2  cm  and  56.6  cm.  For 
aMTADS,  east  and  north  error  standard  deviations  were  27.8  and  36.2  cm. 

The  radial  error  is  the  square  root  of  the  sum  of  the  squares  of  the  north  and  east 
errors.  The  airfield  data  plotted  in  Figure  16  results  in  a  radial  error  histogram  for  each 
system;  these  are  plotted  in  Figure  17.  If  north  and  east  position  errors  are  independent, 
zero-mean  Gaussians  with  equal  variances  in  the  two  dimensions,  the  radial  error 
histogram  should  show  a  Rayleigh  distribution  [8].  That  is  approximately  true  of  the 
aMTADS  errors  in  Figure  17.  The  ORAGS-UV  errors,  on  the  other  hand,  are  far  from 
Rayleigh,  likely  indicating  some  systematic  position  errors.  The  average  aMTADS  radial 
error  in  Figure  17  is  35  cm,  while  the  ORAGS-UV  average  radial  error  is  64  cm.  The 
ORAGS-MV  data,  with  its  large  mean  offsets,  has  an  average  radial  error  of  73  cm. 
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Figure  16.  Location  Error  Scatter  Piots  for  the  Airfieid  Open  Fieid  Area 
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Figure  17.  Radial  Error  Plots  for  Oak  Ridge  Airborne  Geophysics  System-UV  and  Airborne 
Multi-Sensor  Towed  Array  Detection  System  Data  from  the  Airfield  Emplaced  Ordnance 
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In  the  airfield  area  the  total  number  of  deelarations  by  both  sets  of  demonstrators 
was  relatively  small  (608  for  aMTADS  and  845  for  ORAGS-UV).  The  site  area  is  6.02 
heetares,  whieh  translates  to  71  m^  per  piek  for  ORAGS-UV  and  99  m^  for  aMTADS. 
Figure  18  provides  a  plot  of  the  deteetion  deelarations  for  both  demonstrators,  along  with 
the  loeations  of  the  52  emplaeed  UXO  items.  Beeause  of  the  seale  employed  and  the 
symbol  size  required  to  be  able  to  see  the  system  deelarations  and  the  emplaeed  items,  the 
plot  provides  a  distorted  view  of  the  distanee  between  items.  It  is  not  intended  to  provide 
a  quantitative  indieation,  but  instead  a  qualitative  feel  for  the  density  of  deelarations  by 
the  two  systems  and  their  loeations  eompared  with  emplaeed  item  loeations. 

Based  on  the  ehanee  deteetion  model,  all  49  of  the  aMTADS  deteetions  would  be 
deelared  as  true  deteetions.  For  ORAGS,  using  only  the  deelarations  in  eategories  1-4, 
(there  were  no  UXO  deteetions  in  eategories  5  and  6),  34  of  the  35  univariate  and  32  of 
the  33  multivariate  would  be  deelared  true  deteetions,  with  only  one  UXO  deteetion  a 
ehanee  deteetion.  In  the  more  eluttered  areas  that  was  not  the  ease.  For  the  eluttered 
areas,  additional  analysis  is  presented  to  assess  what  portion  of  the  total  number  of 
deteetions  likely  represented  ehanee  deteetions. 

Of  the  three  areas  where  UXO  was  emplaeed,  only  in  the  airfield  area  was  the 
baekground  sueh  that  ealeulation  of  a  deteetion  eurve  vs.  baekground  alarm  rate  made 
sense.  The  dewatering  pond  geometry  made  assessment  of  the  area  that  should  be 
ineluded  for  baekground  diffieult  to  fairly  determine,  and,  with  no  subsequent  reeovery  of 
deteeted  baekground  items,  the  likely  distribution  of  baekground  alarms  between  UXO 
and  elutter  is  unknown.  The  aetive  field  area  was  highly  eluttered,  and,  based  on  later 
reeovery  data  (see  Appendix  B),  many  of  the  baekground  deelarations  in  the  aetive  field 
area  would  have  been  ordnanee  or  ordnanee-related  elutter.  For  those  reasons,  only 
deteetion  of  emplaeed  ordnanee  was  seored  in  the  dewatering  pond  and  aetive  field  areas. 

In  the  dewatering  ponds  area,  ordnanee  emplaeement  was  limited  to  the  large  and 
small  ponds  interior  to  the  field.  Figure  19  [2]  is  a  photograph  of  the  area,  and  Table  9 
provides  the  number  of  ordnanee  emplaeed  and  the  number  deteeted  by  eaeh 
demonstrator.  Figure  20  provides  a  map  of  emplaeed  items  and  the  deteetion  ealls  by 
both  demonstrators.  Only  the  ORAGS-UV  ealls  are  plotted,  but  the  OAGS-MV  algorithm 
produeed  the  same  number  of  deteetions. 
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Figure  18.  Target  Declarations  and  Emplaced  Ordnance  Locations  for  the  Airfield  Open  Field  Area 


Figure  19.  Dewatering  Ponds  Area  [2] 


All  ordnance  items  were  plaeed  in  the  ponds,  and  the  aMTADS  deteetions  are 
localized  to  the  ponds  and  berm  areas  around  the  ponds.  The  ORAGS-UV  deteetions,  on 
the  other  hand,  are  spread  aeross  the  entire  area.  For  that  reason,  we  have  ehosen  not  to 
do  a  ehanee  deteetion  analysis  for  this  area,  as  it  would  be  diffieult  to  apply  in  a  manner 
that  eould  be  justified. 

The  berms  around  the  edges  of  the  ponds  restrieted  the  helieopters’  abilities  to  fly 
elose  to  the  pond  surfaee.  That,  and  beeause  the  targets  were  on  the  pond  bottoms  and 
therefore  at  a  greater  distanee  from  the  sensors  than  for  the  airfield  ease,  made  deteetion 
diffieult.  The  smallest  target,  the  81  mm,  proved  partieularly  diffieult,  although  ORAGS 
did  deteet  1  of  the  16  emplaeed.  When  normalized  for  number  of  ordnanee  of  eaeh  type 
emplaeed,  the  aMTADS  results  in  Table  9  show  a  deteetion  trend  that  increases 
monotonieally  with  ordnanee  size,  as  might  be  expeeted.  ORAGS  deteeted  too  few 
ordnanee  for  any  trend  to  be  evident. 

Table  10  provides  the  eumulative  deteetion  results  as  a  funetion  of  UXO 
likelihood  eall.  The  aMTADS  results  are  based  on  a  total  of  224  deteetion  ealls.  Table  10 
shows  aMTADS  making  all  its  detections  in  the  top  three  UXO  likelihood  eategories  and 
none  in  eategories  4-6.  On  the  surfaee,  that  would  indieate  exeellent  diserimination 
eapability.  In  faet,  that  is  not  the  ease,  as  only  4  of  the  224  deelarations  were  in  eategories 
4-6.  The  ORAGS  univariate  analysis  is  based  on  2,143  deteetion  ealls,  and  the 
multivariate  analysis  is  based  on  2,140  ealls.  Note,  however,  from  Figure  20  that  most  of 
those  deelarations  were  outside  the  pond  areas  where  UXO  were  emplaeed. 
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Table  9.  Ordnance  Detection  Results  for  the  Dewatering  Ponds  for 

Two  Detection  Halos 


Ordnance 

Emplaced 

ORAGS 

aMTADS 

(1  m) 
UV/MV 

(1.5  m) 
UV/MV 

(1  m) 

(1.5  m) 

81  mm 

16 

1/1 

1/1 

0 

0 

105  mm 

25 

1/1 

2/2 

10 

11 

155  mm 

6 

0/0 

0/0 

3 

4 

Total 

47 

2/2 

3/3 

13 

15 

Table  10.  Cumulative  Detection  Probability  as  Function  of  Ordnance  Likelihood  Call  for 

the  Dewatering  Ponds 


UXO 

Likelihood 

ORAGS 

(UV/MV) 

%  detections 

aMTADS 
%  detections 

1  m  halo 

1.5  m  halo 

1  m  halo 

1.5  m  halo 

1 

0/0 

0/0 

19.1 

19.1 

2 

0/2.1 

0/2.1 

25.5 

29.8 

3 

0/2.1 

0/2.1 

27.7 

31.9 

4 

0/4.3 

2.1/4. 3 

27.7 

31.9 

5 

2. 1/4.3 

4.3/6.4 

27.7 

31.9 

6 

4.3/4.3 

6.4/6.4 

27.7 

31.9 

The  final  area  with  emplaeed  ordnanee  was  the  aetive  reeovery  field  (see  Figure 
21  [2]).  There,  ordnanee  was  emplaeed  in  an  open  field,  among  the  phragmites,  and  in  the 
littoral  area.  Note  the  heavy  eoneentrations  of  serap  piles.  As  an  aetive  reeovery  field  for 
indireet  weapons  fire,  this  area  is  very  heavily  eluttered  with  ordnanee  and  ordnanee 
fragments.  Figure  22  provides  a  plot  of  ORAGS-UV  and  aMTADS  anomaly  seleetions, 
along  with  the  loeations  of  emplaeed  items 

Only  81  mm  and  105  mm  shells  were  used  as  emplaeed  ordnanee.  Tables  11  and 
12  provide  target  deteetion  results  by  ordnanee  type  and  as  a  funetion  of  UXO  likelihood 
eall.  Beeause  of  high  residual  elutter,  neither  system  performed  partieularly  well  at 
Aetive  Reeovery  Field,  but  ORAGS  deteeted  twiee  as  many  total  targets  as  aMTADS. 
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Figure  20.  Target  Declarations  and  Emplaced  Ordnance  Locations  for  the  Dewatering  Ponds 


Figure  21.  Active  Recovery  Field  Area  [2] 


Table  11.  Ordnance  Detection  Results  for  Active  Recovery  Field  for  Two  Detection  Halos 


Ordnance 

Emplaced 

ORAGS 

aMTADS 

(1  m) 
UV/MV 

(1.5  m) 
UV/MV 

(1  m) 

(1.5  m) 

81  mm 

32 

2/2 

4/4 

0 

1 

105  mm 

32 

6/6 

6/6 

4 

4 

Total 

64 

8/8 

10/10 

4 

5 

Table  12.  Cumulative  Detection  Probability  as  Function  of  Ordnance  Likelihood  Call  for 

Active  Recovery  Field 


uxo 

Likelihood 

ORAGS 

(UV/MV) 

%  detections 

aMTADS 
%  detections 

1  m  halo 

1.5  m  halo 

1  m  halo 

1.5  m  halo 

1 

0/0 

0/0 

3.1 

4.7 

2 

3.1/0 

3.1/0 

4.7 

6.3 

3 

9.4/0 

9.4/0 

4.7 

6.3 

4 

9.4/1 .6 

9.4/1. 6 

6.3 

7.8 

5 

10.9/9.4 

12.5/9.4 

6.3 

7.8 

6 

12.5/12.5 

15.6/15.6 

6.3 

7.8 
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Figure  22.  Target  Declarations  and  Emplaced  Ordnance  Locations  for  the  Active  Recovery  Field  Area 


The  formula  for  true  matehes  was  applied  to  the  results  from  the  aetive  reeovery 
Field  site  graded  for  the  1.5  m  halo  ease.  The  univariate  results  were  used  for  ORAGS 
beeause  that  algorithm  never  fared  worse  than  the  multivariate  algorithm  and  provided 
better  loeation  aeeuraey.  Table  13  provides  the  results  of  the  analysis.  The  solution  to  the 
equation  is  rounded  to  the  nearest  integer.  A  ealeulation  is  done  for  eaeh  of  the  six 
ordnanee  likelihood  eategories. 


Table  13.  Assessment  of  True  Detection  Statistics  for  the  Active  Recovery  Field  Site 


UXO 

likelihood 

Cumulative 
Number 
of  Declarations 

(ORAGS/aMTADS 

ORAGS-UV 

aMTADS 

Detections 

True 

detections 

Detections 

True 

detections 

1 

117/759 

0 

0 

3 

2 

2 

804/1 628 

2 

1 

4 

2 

3 

2257/1914 

6 

2 

4 

2 

4 

3877/1970 

6 

2 

5 

3 

5 

4676/1996 

8 

3 

5 

3 

6 

4879/1996 

10 

5 

5 

3 

For  the  highly  eluttered  aetive  reeovery  field,  both  systems  deteeted  a  large 
number  of  anomalies:  ORAGS-UV  deelared  4,879  total  deteetions,  and  aMTADS 
deelared  2,968.  Beeause  about  a  thousand  of  the  aMTADS  deelarations  were  in  the 
littoral  region  where  ORAGS  did  not  fly,  only  the  1 ,996  deelarations  shown  on  Figure  22 
were  used  in  the  ehanee  deteetion  analysis.  Although  the  aetive  reeovery  field  is  a  42.1 
heetare  site,  the  large  number  of  deelared  deteetions  resulted  in  ORAGS  ealls  eovering 
8.2%  of  the  total  area  for  a  1.5  m  halo  and  aMTADS  ealls  eovering  3.4%.  Beeause  of  the 
small  pereentages  of  ordnanee  deteeted — espeeially  given  the  pereentage  of  site 
eoverage — it  is  not  surprising  that  50%  of  the  ORAGS  and  40%  of  the  aMTADS 
deteetions  were  likely  by  ehanee. 

In  addition  to  emplaeed  UXO  at  the  aetive  reeovery  field  site,  a  seleeted  number 
of  items  deteeted  by  both  demonstrators  were  reeovered  and  elassified.  IDA  ereated  an 
ordered  dig  list  eomprising  targets  in  the  higher  UXO  likelihood  eategories  (1-3)  for  eaeh 
demonstrator  that  matehed  a  target  loeation  on  the  other  demonstrator’s  list.  Targets  were 
only  ineluded  if  the  loeations  on  the  two  lists  were  within  0.5  m  of  eaeh  other.  Preferenee 
was  given  to  targets  with  a  high  UXO  likelihood  eategory  on  both  lists,  but  a  signifioant 
number  of  ehoiees  differed  signifieantly  in  UXO  likelihood  between  the  two  lists.  To 
reduee  ambiguity,  we  attempted  to  provide  matehed  pairs  that  were  well  separated  from 
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other  detections.  The  average  distance  to  the  next  nearest  detection  was  6  m,  but  a  few 
items  had  nearest  neighbors  less  than  0.5  m  away.  (Some  additional  off-shore  targets 
were  also  dug,  but  we  have  not  included  them  in  this  analysis.)  In  assessing  the  detection 
response  for  the  dug  items,  a  1 .5  m  halo  has  been  used  relative  to  the  actual  dug  location 
of  the  excavated  items,  not  the  originally  provided  coordinates.  In  scoring  discrimination, 
the  recovered  items  have  been  classified  as  intact  ordnance  (O),  ordnance-related  (OR), 
or  clutter  (C).  Some  items  were  listed  as  having  unreliable  recovery  data  or  had  no 
description.  Those  were  also  not  included  in  the  analysis.  Figure  23  shows  the  dug  item 
locations,  and  Appendix  B  provides  a  list  of  the  items,  with  locations  relative  to  the 
coordinates  of  Figure  23.  Note  the  heavy  preponderance  of  ordnance  and  ordnance- 
related  items  in  the  list,  where  we  have  classified  46  items  as  ordnance,  56  as  ordnance- 
related,  and  only  16  as  clutter. 

Because  all  the  dug  items  were  chosen  from  items  that  appeared  on  both  detection 
lists,  we  would  expect  the  probability  of  detection  to  be  near  unity,  and  that  is  the  case. 
Both  systems  had  a  detection  within  a  1.5  m  halo  for  115  of  the  118  items.  The  two 
systems  missed  the  same  3  items  (2  ordnance  and  1  clutter),  in  every  case  because  the 
actual  coordinates  of  the  dug  item  were  more  than  1.5  m  from  the  dig  list  coordinates 
provided  and  were  not  within  1 .5  m  of  a  declaration  on  either  list. 

Figures  24  and  25  provide  ROC  curves,  based  solely  on  the  items  of  Appendix  B. 
Two  cases  have  been  calculated:  (1)  only  intact  ordnance  counted  as  UXO  and  (2)  intact 
ordnance  and  ordnance-related  items  counted  as  UXO.  Figure  24  shows  ROC  cures  for 
the  first  case.  Here,  ordnance-related  clutter  and  clutter  are  included  in  the  false-alarm 
category.  Figure  25  provides  ROC  curves  for  the  second  case.  Here,  false  alarms  are  the 
non-ordnance-related  clutter.  For  both  cases,  all  the  items  matched  in  the  Appendix  B  list 
were  matched  by  aMTADS  category  1-3  detections,  and  ORAGS-UV  had  detections  in 
all  six  categories.  aMTADS  performs  somewhat  better  than  ORAGS-UV  in  the  first  case, 
but  it  is  worse  for  the  second  case.  In  neither  case  does  the  performance  ever  rise 
substantially  above  the  chance  diagonal.  The  data  likely  only  illustrate  the  difficulty  of 
classification  for  an  airborne  magnetometer,  with  its  wide  single-sensor  footprint,  in  a 
highly  cluttered  area  like  the  active  recovery  field. 
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Figure  23.  Target  Declarations  and  Recovered  Target  Locations  for  the  Active  Recovery  Field  Area 


Only  Intact  Ordnance  counted  as  UXO 
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Figure  24.  Receiver  Operating  Characteristic  Curves  for  the  Active  Recovery  Field  Dug 
Items  Where  Only  Intact  Ordnance  is  Counted  Toward  Probabiiity  of  Correct  Ciassification 


Ordnance-Related  Items  counted  as  UXO 


Prob.  of  False  Alarm 


Figure  25.  Receiver  Operating  Characteristic  Curves  for  the  Active  Recovery  Fieid  Dug 
Items  Where  Intact  Ordnance  and  Ordnance-Related  Clutter  are  Counted  Toward 
Probability  of  Correct  Classification 


11-35 


III.  FINDINGS  AND  CONCLUSIONS 


A,  FINDINGS 

The  demonstrations  analyzed  in  this  report  were  designed  to  test  the  deteetion  and 
elassifieation  eapabilities  of  airborne  magnetometer  systems  over  a  wide  range  of  terrain 
types  and  levels  of  baekground  elutter.  UXO  was  emplaeed  in  seleeted  areas  to  provide 
known  items  against  whieh  system  performanee  eould  be  judged.  This  report  foeuses  on 
overall  performanee,  whieh  is  a  funetion  of  a  number  of  faetors.  These  inelude,  as  a 
minimum,  inherent  sensor  sensitivity,  sensor  footprint,  proximity  to  the  target,  spatial 
sampling  densities  in  down-  and  eross-traek,  target  loeation  aeeuraey,  and  signal 
proeessing.  Signal  proeessing  eomprises  prefiltering  for  baekground  variation  removal, 
interferenee  reduetion  and  sensor  output  leveling,  target  loeation  (whether  manual  or 
automatie),  and  elassifieation  proeessing. 

These  two  systems  employ  the  same  model  eesium  vapor  magnetometers  earried 
on  identieal  helieopters.  Although  the  details  of  the  eleetronies  differ  between  the 
systems,  we  would  expeet  their  inherent  signal  sensitivities  to  be  similar,  and  that  was  the 
ease  for  the  ealibration  targets  and  airfield  targets  analyzed  in  this  report.  While  the 
aMTADS  typieally  produeed  a  signal  level  two  to  five  times  that  of  ORAGS,  this  ean  be 
explained  by  the  differenee  in  helieopter  altitudes  flown  during  the  survey. 

The  inherent  noise  floors  for  the  two  systems,  inferred  from  the  high-altitude  data, 
also  appear  to  be  very  similar,  but  eurrent  ORAGS  proeessing  provides  four  or  five  times 
more  baekground  variation  over  quiet  areas  in  low-altitude  flight  than  does  aMTADS. 
The  eombination  of  the  lower  flight  altitudes  and  better  noise  suppression  gave  aMTADS 
an  average  SNR  advantage  against  eomparable  targets  of  a  faetor  of  10  in  the  ealibration 
area  and  17  in  the  open  field  area  of  the  airfield. 

The  other  two  seored  areas  presented  mueh  more  diffieult  problems  for  both 
systems  than  did  the  fiat,  relatively  uneluttered  airfield  area.  The  dewatering  ponds  would 
have  been  ehallenging  under  the  best  of  eireumstanees  beeause  of  the  separation  between 
the  sensors  and  the  targets.  Only  10  of  the  48  emplaeed  ordnanee  items  were  a  foot  or 
less  below  the  water  surfaee,  with  3 1  of  the  items  3  feet  deep  or  more,  and  1 8  of  the  items 
6  or  more  feet  deep.  In  addition,  berms  around  the  ponds  made  it  diffieult  for  the  pilots  to 
fly  elose  to  the  surfaee,  partieularly  near  the  edges  of  the  ponds.  Qualitative  reports  from 
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the  demonstrators  indieate  that  aMTADS  was  able  to  fly  lower  over  the  ponds  than 
ORAGS — this  shows  in  the  deteetion  results,  where  DRAGS  deteeted  only  3  emplaeed 
items  but  aMTADS  deteeted  15.  However,  10  of  the  15  aMTADS  deteetions  were  2  feet 
deep  or  less  and  the  1  very  deep  item  (a  155  mm  shell,  5.5  feet  deep)  and  1  item  3  feet 
deep  were  likely  ehanee  matehes,  as  the  aMTADS  loeation  errors  were  more  than  1  m.  In 
normal  land-elearanee  operations,  deep  UXO  items  are  also  typieally  massive,  so  that 
signature  somewhat  offsets  the  distanee  falloff.  In  the  water,  that  is  not  the  ease,  and 
airborne  or  surfaee  magnetometer  systems  are  likely  to  be  effeetive  only  for  very  shallow 
bodies  of  water  or  where  only  very  large  targets  are  of  interest. 

The  aetive  reeovery  field  is  a  very  eluttered  site,  and  the  results  there  eonfirm  the 
generally  aeknowledged  feeling  that  effeetive  elearanee  of  buried  items  in  sueh  situations 
requires  elearanee  of  surfaee  elutter  beforehand.  Neither  system  did  well  in  deteeting 
emplaeed  items,  and  around  half  the  items  deteeted  by  both  systems  were  likely  deteeted 
by  ehanee. 

B,  CONCLUSIONS 

Several  eonelusions  ean  be  drawn  from  the  results  of  these  demonstrations: 

•  If  maximum  sensitivity  is  desired,  it  is  important  that  the  system  design  allow 
very  low-level  flight  in  areas  where  that  is  possible.  Careful  attention  to 
redueing  apparent  spatial  baekground  variations  eaused  by  platform 
dynamies  effeets  is  also  erueial  in  obtaining  maximum  deteetion  sensitivity. 
In  the  very  benign  airfield  terrain  and  baekground,  aMTADS  deteeted  49  of 
52  emplaeed  items,  ineluding  all  three  60  mm  mortars  and  eighteen  of 
twenty-one  81  mm  mortars,  but  ORAGS,  flying  about  a  meter  higher  and  3 
times  as  fast,  and  with  4  times  the  baekground  standard  deviation,  deteeted 
14  fewer  emplaeed  items,  missing  one  of  the  60  mm,  1 1  of  the  81  mm,  and  5 
of  the  105  mm. 

•  Sueeessful  deteetion  of  UXO  targets  in  bodies  of  water  will  require 
knowledge  of  the  burial  depth  and  types  of  ordnanee  expeeted.  Helieopter 
systems  sueh  as  these  already  have  the  disadvantage  of  a  sensor  that  is  a 
meter  or  two  above  the  surfaee.  Beeause  of  the  1 /(distanee)^  signal  falloff, 
even  moderate  target  depths  make  the  deteetion  of  all  but  large  targets 
marginal. 

•  Neither  system  shows  signifreant  individual  target-deteetion  eapability  in 
highly  eluttered  areas. 

•  Target  loeation  aeeuraey  is  important  for  unambiguous  target  reeovery. 
aMTADS  demonstrates  that  average  radial  position  errors  of  less  than  40  em 
are  possible  from  an  airborne  system  using  real-time,  differential  GPS. 
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However,  obtaining  that  degree  of  aeouraey  requires  reliable  sensing  and 
preeise  aeeounting  for  helieopter  dynamie  effeets  on  sensor  position. 

Aeeurate  automatie  deteetion  and  elassifioation  algorithms  would  be  of 
signifieant  benefit  for  systems,  sueh  as  the  ones  demonstrated  here,  that  are 
eapable  of  surveying  hundreds  of  aeres  per  day.  Current  automatie 
algorithms,  however,  do  not  perform  as  reliably  as  human  analysts. 
Partieularly  in  the  deteetion  phase,  they  result  in  many  more  anomalies  that 
would  have  to  be  dug,  without  adequate  diserimination  eapability  to  make  up 
for  the  additional  deteetions. 

Current-generation  helieopter-borne  magnetometer  systems  are  eapable  of 
reliably  deteeting  areas  of  high  ferrous  elutter/UXO  density.  It  appears  they 
are  not  suitable  for  general  survey  use  to  deteet  individual  ordnanee  items, 
exeept  in  speeial  eonditions  that  provide  a  eombination  of  benign  terrain,  low 
baekground  elutter,  and  appropriate  UXO  sizes  (>60  mm).  Even  then,  lower 
and  slower  flight  than  indieated  in  system  speeifieations  will  likely  be 
required  to  aehieve  adequate  deteetion  performanee. 
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ACRONYMS 


aMTADS 

Airborne  Multi-Sensor  Towed  Array  Deteetion  System 

APG 

Aberdeen  Proving  Ground 

ESTCP 

Environmental  Seeurity  Teehnology  Certifieation  Program 

FFT 

fast  Fourier  transform 

GPS 

Global  Positioning  System 

HAF 

height  above  ellipsoid 

IDA 

Institute  for  Defense  Analyses 

MTADS 

Multi-Sensor  Towed  Array  Deteetion  System 

NRF 

Naval  Researeh  Fab  oratory 

DRAGS 

Oak  Ridge  Airborne  Geophysies  System 

ORNF 

Oak  Ridge  National  Faboratory 

Pd 

probability  of  deteetion 

ROC 

reeeiver  operating  eharaeteristie 

SFRDP 

Strategie  Environmental  Researeh  and  Development 
Program 

SNR 

signal-to-noise  ratio 

STDFV 

Standard  deviation 

UTM 

Universal  Transverse  Mereator 

UXO 

unexploded  ordnanee 
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APPENDIX  A 

CHANCE  DETECTION  ANALYSIS 


In  analyzing  detection  performance,  evaluators  long  ago  realized  that  some 
number  of  detections  were  likely  to  be  due  to  chance.  That  is,  a  certain  number  of 
detections  that  are  graded  as  true  detections  are  not  actually  associated  with  the  item 
assumed  to  be  detected.  To  illustrate  the  point,  if  N  detection  calls  were  randomly 
distributed  over  a  test  plot  with  area  A,  a  demonstrator  might  expect,  just  based  on 
chance,  to  achieve  an  apparent  probability  of  detection  approximately  equal  to  NnRhIA, 
where  Rh  is  the  radius  of  the  halo  used.  A  more  sophisticated  analysis  is  developed  in 
Reference  7  to  assess  the  number  of  these  “chance”  detections.  The  assumption  made  is 
that  the  survey  instrument  has  inherent  random  errors  in  locating  targets  that  can  be 
described  by  zero-mean  and  equal  variance  Gaussian  distributions  in  two  orthogonal 
dimensions.  For  that  assumption,  we  would  expect  the  probability  density  function  for  the 
radial  error  to  be  Rayleigh  and  the  probability  of  detecting  an  object  within  a  radius  Rh  of 
the  declared  location  would  be  given  by  [8]: 


where  a  is  the  location  error  standard  deviation  in  the  east  and  north  directions.  If  the 
halo  is  substantially  larger  than  the  error  standard  deviation,  then  the  probability  that  a 
detected  target  lies  inside  the  halo  is  essentially  unity.  For  example,  for  an  Rhio  =  2,  the 
probability  is  94%,  and  for  Rhio  =  2.5,  it  is  99%.  From  the  airfield  emplaced  object 
detections,  the  worst  dimension  location  error  standard  deviations  were  57  cm  for 
ORAGS-UV  and  37  cm  for  aMTADS.  The  1.5  m  standard  halo  used  in  the  APG  analysis 
is  more  than  2.5  times  the  ORAGS  standard  deviation,  and  so  approximating  the 
probability  as  unity  is  reasonable.  Applying  that  approximation  to  the  analysis  gives  the 
number  of  “true”  detections  as 

T  =  M -U-F -a 

where 

T  =  number  of  true  matches 
U  =  true  number  of  missed  detections  =  B  -  T 
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B  =  number  of  items  buried 
F  =  true  number  of  false  deteetions  =  D  -  T 
D  =  number  of  target  deelarations 
a  =  (area  of  halo)/(area  of  site) 

M  =  Sum  of  true  +  lueky  matehes 


This  results  in  a  quadratie  in  T  whose  solution  is 
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ya  J 
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Curves  of  true  deteetions  as  a  function  of  halo  should  increase  as  halo  size  increases  and 
reach  an  asymptotic  value  that  is  indicative  of  sensor  detection  performance.  For  a  sensor 
that  produces  small  position  estimation  errors,  the  curve  rises  rapidly  to  its  final  value, 
but  for  a  sensor  with  less  accurate  positioning,  the  curve  reaches  asymptote  more  slowly. 
When  actual  data  are  plotted,  however,  curves  of  true  detections  reach  a  peak  value  and 
eventually  start  to  decrease  with  increasing  halo.  The  apparent  decrease  is  due  to  a 
violation  of  the  assumptions  under  which  the  above  equations  were  derived.  The  analysis 
assumes  that  declarations  are  widely  enough  separated  so  that  halos  never  overlap  and 
portions  of  halos  never  lie  outside  the  analysis  area.  At  some  point,  those  assumptions 
break  down  and  the  number  of  chance  detections  does  not  continue  to  rise  proportionally 
to  the  halo  area,  so  more  chance  detections  are  calculated  than  actually  occur.  However, 
until  the  assumptions  are  violated,  the  above  development  gives  a  good  estimate  of  true 
versus  chance  detections. 
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APPENDIX  B 

TABLE  OF  ACTIVE  RECOVERY  FIELD  DUG  ITEMS 


East 

North 

Description 

Class.* 

223.41 

424.19 

8-inch  projectile,  unfired  (salute  rd) 

o 

319.11 

393.71 

2.75  in  rocket  warhead  fired,  unfuzed 

o 

306.03 

422.30 

155-mm  projectile  fired,  fuzed  3  fragments 

o 

208.98 

401.04 

1/2  of  105-mm  casing 

o 

556.49 

455.45 

155-mm  projectile,  fuzed,  fired 

o 

275.28 

628.87 

90-mm  projectile,  fuzed,  fired 

o 

109.63 

390.62 

1/2  of  90-mm  casing 

o 

479.18 

736.80 

Projectile  frag  (90-mm)  /w  fuze 

o 

262.99 

612.22 

120-mm  projectile  fuzed,  fired 

o 

110.96 

479.15 

5-inch  projectile  fired,  unfuzed 

o 

121.42 

477.00 

155-mm  projectile  fuzed,  fired 

o 

193.45 

328.07 

90-mm  projectile 

o 

120.64 

472.48 

105mm  projectile,  fired,  fuzed 

o 

369.27 

822.32 

155-mm  projectile,  fuzed,  fired 

o 

247.81 

500.27 

155-mm  projectile  fuzed,  fired 

o 

484.97 

446.24 

6-inch  projectile,  fuzed,  fired 

o 

617.72 

843.72 

90-mm  projectile  casing,  unfuzed 

o 

543.62 

687.75 

Low-order  90  or  105  mm  projectile 

o 

266.17 

412.38 

5-inch  projectile,  unfuzed,  fired 

o 

606.78 

795.06 

175-mm  projectile,  fuzed,  fired 

o 

226.16 

679.12 

90-mm  projectile,  fuzed,  fired 

o 

256.49 

705.08 

1 55-projectile 

o 

123.01 

541.92 

Fuzed  155-mm  projectile 

0 

522.12 

813.25 

75-mm  projectile,  fuzed,  fired 

o 

161.44 

317.91 

155-mm  projectile,  fuzed,  fired 

o 

186.54 

577.60 

155-mm  projectile  unfuzed  fired 

o 

551.30 

697.65 

75-mm  projectile,  fuzed,  fired 

o 

372.19 

552.56 

120-mm  projectile  fuzed,  fired 

o 

579.39 

816.16 

75-mm  projectile,  fuzed,  fired 

o 

348.12 

692.21 

14-in  fuzed  projectile 

o 

140.14 

525.22 

120-mm  projectile  fuzed,  fired 

o 

478.76 

938.70 

155-mm  projectile,  fuzed,  fired 

o 

218.83 

532.54 

155-mm  projectile  unfuzed  fired 

o 

339.30 

646.63 

155-mm  projectile,  fuzed,  fired 

o 

259.46 

473.42 

90-mm  AP  round  fired  large  piece  of  scrap  metal 

o 

581.53 

958.10 

155-mm  fired  fuzed 

o 

277.83 

647.49 

90-mm  projectile,  unfuzed  fired  large  fragment 

o 

302.18 

714.75 

90-mm  projectile,  fuzed 

o 

406.45 

555.34 

120-mm  projectile  fuzed,  fired 

o 

251.01 

411.91 

106-mm  RAP  round 

o 

98.04 

538.35 

90-mm  projectile,  fuzed  fired 

o 

142.98 

373.49 

240-mm  projectile,  fuzed,  fired 

o 

597.03 

705.65 

8-in  projectile,  fuzed,  fired 

o 

437.16 

840.02 

155-mm  projectile,  fuzed,  fired 

o 

404.67 

667.36 

155-mm  projectile,  fuzed,  fired 

o 

446.50 

479.29 

8-inch  projectile 

o 
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East 

North 

Description 

Class.* 

281.98 

466.83 

Fragment 

OR 

431.47 

477.14 

Fragment 

OR 

418.39 

823.22 

Fragment 

OR 

193.12 

526.42 

Fragment 

OR 

494.58 

709.45 

Fragment 

OR 

458.00 

495.14 

Fragment 

OR 

164.95 

509.67 

Fragments 

OR 

426.15 

841.09 

Large  piece  of  fragment 

OR 

165.95 

529.83 

Fragment 

OR 

545.40 

769.32 

Projectile  fragments 

OR 

383.05 

782.73 

Bomb  plug 

OR 

212.03 

477.58 

Fragment 

OR 

542.59 

802.26 

Fragment 

OR 

615.89 

778.97 

Fragments  from  90-mm  projectile 

OR 

391.97 

665.92 

Large,  thin-wall  (bomb?)  fragment;  unable  to  recover 

OR 

192.80 

362.00 

Fragment 

OR 

316.63 

425.90 

105-mm  fragment 

OR 

577.30 

851.89 

Fragment 

OR 

561.10 

866.51 

Fragment 

OR 

263.50 

419.50 

Fragment 

OR 

216.37 

401.33 

Fragment 

OR 

370.46 

478.64 

Fragment 

OR 

532.23 

902.46 

Fragments 

OR 

142.81 

408.81 

Rod  Fragments 

OR 

538.23 

827.30 

155-mm  fragment 

OR 

194.12 

405.00 

Fragment 

OR 

390.34 

802.84 

Fragments 

OR 

407.31 

800.08 

Fragments 

OR 

443.29 

828.84 

Steel  fragment 

OR 

203.26 

391.47 

Fragments 

OR 

363.07 

782.14 

Fragment 

OR 

352.79 

419.07 

Fragment 

OR 

206.64 

504.73 

Fragments  WP  projectile 

OR 

142.86 

358.70 

Fragments  (low  order  det.) 

OR 

383.51 

810.01 

Fragment 

OR 

214.29 

471.37 

Fragment 

OR 

236.34 

437.58 

Fragment 

OR 

299.94 

338.36 

Fragments 

OR 

316.67 

681.99 

Suspect  ammo  burial  pit  -  pipe  and  fragments 

OR 

523.79 

430.45 

Fragments  and  rebar 

OR 

176.14 

613.47 

Fragments 

OR 

515.70 

805.46 

Fragment 

OR 

158.23 

349.03 

155-mm  projectile  frag 

OR 

469.78 

817.00 

Fragment 

OR 

485.03 

662.36 

Fragment 

OR 

548.66 

702.03 

Bomb  fragment 

OR 

339.31 

411.05 

Fragments 

OR 

213.59 

509.98 

Fragments  (3)  possible  WP 

OR 

230.32 

422.72 

Frag,  base  of  155 

OR 

629.01 

531.30 

Fragments 

OR 

325.27 

422.06 

Fragment 

OR 

550.58 

398.03 

Small  fragments 

OR 

402.80 

601.80 

Fragments 

OR 

564.24 

922.80 

Fragments 

OR 

467.91 

494.34 

Fragment 

OR 

173.05 

214.61 

Fragment 

OR 
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East 

North 

Description 

Class.* 

462.93 

693.36 

Cylinder 

c 

360.27 

695.46 

Steel  plate 

c 

323.88 

623.51 

Large  piece  of  angle  iron 

c 

601.84 

717.23 

Railroad  spike 

c 

465.53 

503.12 

Railroad  rail  on  end 

steel  core  ground  rod,  approx  0.6  meters  bent  to  ground  surface 

c 

217.09 

637.17 

-1.2  m  in  ground 

c 

465.27 

831.50 

Steel  plate  ring 

c 

282.40 

377.21 

Steel  plate 

c 

445.86 

509.04 

Rebar  in  concrete 

c 

414.85 

472.83 

Pipe  and  ring 

Household  waste  pile,  metal  pitcher,  cups,  wash  buckets,  misc. 

c 

154.26 

227.10 

scrap  metal 

c 

580.51 

672.81 

Large  piece  of  4"  (102-mm)  angle  iron 

c 

517.61 

761.81 

Two  metal  rods 

c 

490.34 

831.32 

Scrap  metal 

c 

101.77 

334.25 

Scrap  metal 

c 

162.63 

260.44 

Steel  plate 

c 

*  O  =  Ordnance 

OR  =  Ordnance  Related 
C  =  Clutter 
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